Parsing messages with multiple data formats

ABSTRACT

Provided are a method for parsing a bit stream including multiple data formats, and an apparatus and computer program including a set of parsers and parser-selection and invocation capabilities for handling parsing of multiple data formats. A first parser is selected and invoked to handle a first formatted component of the bit stream, and this selected parser selects and invokes a next parser which is capable of handling a differently formatted next component of the bit stream. This is differentiated from systems which rely on a single generic parser or a single high-level parser selection process, and is especially advantageous when parsing messages to enable message processing in systems in which a message can include multiple different, nested data formats.

This application is a continuation of application Ser. No. 11/757,127,filed Jun. 1, 2007, status pending which in turn is a continuation ofapplication Ser. No. 10/076,358, filed Feb. 14, 2002, status allowed.

FIELD OF INVENTION

The present invention relates to parsing of data, such as in a messagingsystem in which messages may include a number of different data formats.

BACKGROUND

Parsing of data typically involves an initial lexical analysis stepwhich involves breaking up the input data into logically separatecomponents, such as field names, constants, operators (the lexicalanalyser outputs a string of ‘tokens’) and then a syntactic analysisstep which involves processing the tokens to determine the syntacticstructure of the data as defined by a grammar. A lexical analyser mayalso remove redundant spaces and may deal with character-set mappings(e.g. replacing upper case letters with lower case). The term parser isused to refer to a program which performs such analysis. The output of asyntax analyser may be a syntax tree (also known as a parse tree) whichrepresents the syntactic structure of the string of data being parsed.Parsing is well known in the context of compilers, but is a requiredstep for many data processing operations. For example, a messageprocessing program may need to parse an input message to model themessage structure before it can begin processing the message—for exampleto understand the structure and format of the message before performingformat conversion. This may include separating a message descriptor intoa set of fields comprising name-value pairs so that the different valuesin the named fields can be processed, and similarly separating a streamof bits comprising the message data into name-value pairs so that thedata can be processed.

It is now very common for a computing network to integrate manyheterogeneous systems, and individual messages sent across thesenetworks may include different data formats within headers inserted bythe different systems through which the message passes. It is thereforeimportant for a network-wide messaging service to be capable of handlinga number of different data formats within a single message, to supportthe increasing requirement for business and system integration. It is afeature of some existing messaging products to be able to parse anincoming message which includes a number of different format components,splitting the message into its differently formatted constituent partsand separately parsing these parts to generate an output message forfurther processing.

In the past, these messaging products have relied on predefined messageformats and either a generic parser or a single parser-selection processwhich has access to a repository of message formats. Either the genericparser analyses all components itself or a process scans through themessage to identify the differently formatted components, comparing theidentified formats with those in the repository, and then a selectorassigns each component to a specific parser which is capable ofperforming syntactic analysis for that component.

This approach has proven satisfactory for situations in which only asmall number of formats are possible and the format and sequence of allthe message components is known in advance, since this knowledge enablesthe selection of appropriate parsers for the different components. Itmay also be possible in some cases, IC although inefficient, to rely ona generic parser to perform an initial analysis and resultingparser-selection before the main syntactic analysis begins. However,this would require the generic parser to be capable of analysing alldata formats within each message so that the first scan of the messagecould break the message into components and could provide theinformation to determine which specific parser should handle whichcomponents. It may be very difficult to implement such capability withina single generic parser. Secondly, separating syntactic analysis intoseparate first and second steps would entail processing delays and wouldtend to duplicate some of the analysis.

An alternative to this initial step of a generic parser analysing thewhole message is for individual selected parsers to perform syntacticanalysis of specific parts of a message, including identifying the dataformat or data type of the next message component in the sequence ofbytes which makes up the message. In this case, a parser-selectionprocess can identify the format of a first component and select a firstspecific parser; the selected parser can then parse this firstcomponent, identify the format or type of the next component and sendthis format or type information to the parser-selector to invoke anotherspecific parser for this next message component. As noted above, suchsolutions rely on knowledge of predefined message formats andpredictable sequences of message components. This has provensatisfactory for handling differently formatted message headers, if theselected parsers are given the knowledge of which field to read todetermine the format of the next message component, and if the datawithin the body of the message has a single format such that a singleparser can parse the entire contents of the message body. The class nameof the required parser can be included in a format field of a messagedescriptor or another header of the message, and this can be read toselect and then invoke the appropriate parser.

A problem arises with the above solution when a message body includesmultiple nested data formats, since then the reliance on the singleparser-selector to call the appropriate program at the required timeinvolves an excessive number of communication flows between the specificselected parsers and the parser-selector. It also requires theparser-selector to be able to invoke a suitable specific parser for allformats and for unpredictable nested structures, such that the selectorneeds a detailed knowledge of an ever increasing number of formats. Theproblems of this approach will become clear in future as the number ofmessage data formats and the complexity of message contents increaseswith increasing systems and business integration.

Thus, there remains a requirement for an efficient solution to parsingof messages which include multiple data formats, especially when thedifferent formats can be nested within one another and the structure ofthe message is not known in advance of its receipt and analysis by amessaging program. The problems of known solutions are especially acutefor messages in which either individual components or the sequence ofcomponents do not have a predictable structure, since then a messageanalysis is required as part of the run-time operation of the messagingprogram before it is possible to select a parser to analyse a nextcomponent of the message.

SUMMARY OF INVENTION

In a first aspect, the present invention provides a data processingapparatus for processing messages which may contain a plurality of dataformats, the apparatus including a set of selectable parsers, eachadapted for analysing a specific set of one or more data formats,wherein a plurality of said set of parsers each include: means forparsing a first component of a message, means for identifying the dataformat of a second component of the message, and means, responsive tosaid identification, for selecting another one of said set of parsersand for invoking the selected parser to parse the second messagecomponent.

This preferably includes the capability of a first selected parser, towhich a portion of a message has been allocated for parsing, to identifysubcomponents within this message portion which should be allocated to adifferent parser and to select and invoke a parser using subcomponentformat information within the message. The first selected parserpreferably passes only a specific subcomponent to the different parserand keeps control for analysing format information for othersubcomponents to select and invoke an appropriate parser for thosesubcomponents.

This invention changes the distribution of responsibilities comparedwith existing parsing solutions, producing an hierarchical tree ofparsers to handle nested-format messages instead of the horizontal chainof parsers controlled by a single selector process (which is the resultof known solutions). The depth of the hierarchy can be determined inresponse to the structure of a received message rather than being hardcoded into the parser program. The invention mitigates theinefficiencies of solutions which rely on a single generic parser tohandle all of the analysis of message components or to perform a partialanalysis to determine which specific parsers to use. It also mitigatesthe inefficiencies of solutions in which a plurality of parsers arealways required to call a generic parser-selector to invoke anotherspecific parser.

Since an analysis step performed by a generic parser cannot be optimisedfor all data formats, messaging systems which rely on generic parserswill be unable to efficiently handle multiple-format messages in aheterogeneous network. As systems integration requirements and thenumber of different message formats increase, the inadequacies of suchsystems will become clear. Similarly, reliance on a singleparser-selector process will prove inadequate as the number of messagedata formats and the possibility of nested-formats and unpredictablestructures within the body of a message increases.

Hence there are significant benefits in the solution of the presentinvention which enables specific parsers to perform the selection andinstantiation of a next parser without having to make a call to ageneric parser or parser-selector to perform this instantiation. Theinvention enables a reduction in the cost and potential delays ofinter-process communication flows and avoids having to continuallyupdate the generic parser or parser-selector to provide capabilities tocope with new formats and structures. Nevertheless, the solution of thepresent invention is in contrast to the current technological trend ofrelying on a single generic parser or a single specialised parserselection process. The invention requires a recognition that thelimitations of current systems will result in significant problems whenparsing complex nested-format messages, and a recognition that changingthe structure of and distribution of responsibilities within a parsingmechanism by adding parser-selection and parser-invocation capabilitiesto specific selectable parsers will provide significant benefits whichoutweigh the costs.

A specific example of the problems of known prior art solutions ariseswhen a message to be parsed contains a component in SAP AG's IDoc format(IDoc is the short-form name of the ‘Intermediate Documents’ exchangedwithin SAP's R/3 system, and the name is also used to describe the dataformat of these documents. SAP and R/3 are trademarks of SAP AG). AnIDoc component contains a control structure component (DC) and avariable number of data components (DD) and it is only possible todetermine the number of DDs during parsing. There may also be multipleIDocs within a single message stream. A typical generic parser would notbe capable of separating an IDoc into its constituent parts andanalysing those parts. Similarly, it would be inefficient to rely on asingle generic parser-selection process to assign the DC and DDs todifferent parsers. A preferred embodiment of the present inventionallows a specific IDOC parser to be selected in response to a parser ofa message descriptor or header identifying that the next component hasthe IDoc format. The IDOC parser can then analyse the DC component andselect and instantiate a further parser for handling each of the DDcomponents. After each DD component is handed to a newly instantiatedparser, the IDOC parser retains responsibility for identifying the nextcomponent (whether a PD or another DC) and selecting the appropriateparser. This ability to instantiate a parser to handle a subcomponent ofan allocated portion of a message is a significant difference fromcontrol always being returned to a single parser-selector for eachcomponent.

As well as enabling incoming IDoc data to be represented to a messageprocessing program in a format that the program can understand andmanipulate, the IDOC parser according to a preferred embodiment of thepresent invention also supports creation of output messages in the SAPIDoc format. This two-way format conversion capability supportsintegration between SAP systems and a message processing program whichincludes the IDOC parser.

To implement IDoc parsing capability within a single genericparser-selector, and to add similarly specific functionality to thesingle selector for other data formats such as SWIFT messages, wouldresult in excessive program complexity and require repeated re-coding ofthe selector to support new message types. The SWIFT message format,defined by the Society for Worldwide Interbank FinancialTelecommunication, is another example of a message format where thesequence of components is not fully known in advance.

As the requirements for systems and business integration extend to coveradditional systems, and as new data formats are created, the presentinvention provides an extensible solution to which new parsers can beadded which have integrated parser-selection capabilities. This iseasier to update and maintain than known solutions in which the newcapabilities have to be added to a generic parser or a singleparser-selector.

In a second aspect, the invention provides a method of parsing a messagecontaining a plurality of data formats, the method comprising:identifying the data format of a first component of the message;responsive to said identification, selecting and invoking a first parserto parse the first component; and using said first parser to identifythe data format of a second component of the message and, responsive tosaid identification, using said first parser to select and invoke asecond parser for parsing the second message component.

In a preferred method, the second message component can be asubcomponent of the first component, and the first selected parser isable to assign a specific chunk of a message bit stream (correspondingto this second subcomponent) to the second parser while retainingcontrol for selecting and invoking a parser for subsequent messagecomponents. In the example of IDoc message components mentioned above,the IDOC parser assigns a DD to a specific parser while retainingresponsibility for identifying a next message component (whether a DC ora specific format DD) and for selecting and invoking a next parser (fora DD) or parsing the next component itself (in the case of a DC).

In a third aspect, the invention provides a message processing systemincluding a parser selector and a set of selectable parsers, eachselectable parser being adapted for analysing a respective set ofmessage data formats, wherein said set of selectable parsers are eachselectable in response to identifying a message data format within therespective set, and wherein at least one of said selectable parsersincludes: means for parsing a first component of a message having amessage data format within the respective set; means for identifying thedata format of a second component of the message; and means, responsiveto said identification, for selecting another one of said set of parsersand for invoking the selected parser to parse the second messagecomponent.

The above-described selectable parser preferably comprises an IDOCparser for parsing message data in the IDoc format, including means forparsing a DC component of an IDoc, means for identifying a DD component,and means for selecting and invoking another parser for parsing theidentified DD component.

In a fourth aspect, the invention provides a computer program forcontrolling the operation of a data processing apparatus on which itruns to perform a method as described above. The computer program may bemade available as a program product comprising program code recorded ona computer readable recording medium.

BRIEF DESCRIPTION OF DRAWINGS

Preferred embodiments of the invention will now be described in moredetail, by way of example, with reference to the accompanying drawingsin which:

FIG. 1 shows a message broker within a messaging network, including aparsing solution according to an embodiment of the present invention;

FIG. 2 shows a simple example message structure;

FIG. 3 is a schematic representation of message flows between a SAP R/3system and a message broker such is as IBM's MQSeries Integrator broker;

FIG. 4 is a representation of the hierarchical parser's view of amessage which includes an IDoc; and

FIG. 5 is a representation of a sequence of steps according to anembodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 is a schematic representation of a messaging network in which anumber of application programs 10 are communicating via messagingmanager programs 20 which provide message delivery services and via amessage broker 30 which implements a set of rules to provide messagerouting and message transformation (and message creation in response tocertain received messages). The message broker 30 and messaging managerprograms 30 enable diverse applications to exchange information inunlike forms—the brokers handling the processing required for theinformation to arrive in the desired place in the correct format and themessaging manager programs handling the complexities of networkcommunications to provide assured delivery of messages across thenetwork between different computer systems. Such messaging services areprovided by the commercially available MQSeries family of products fromInternational Business Machines Corporation (MQSeries is a trademark ofIBM Corporation).

IBM's MQSeries Integrator products support business and applicationintegration, using message brokers 30 to manage the flow of informationbetween applications. The brokers are a resource which hosts andcontrols message flows 40 which implement business processes.Applications send messages to a message flow, which is a sequence ofmessage processing nodes that each implement rules (a set of processingactions), to process the messages and generate one or more outputs forsending to target applications using MQSeries messaging and queuingcommunications. IBM's MQSeries Integrator products include a ControlCentre which provides a user interface for creating message flows byselecting and connecting message processing nodes. IBM's MQSeriesIntegrator products are described in more detail, for example, inMQSeries Integrator v2.0.1 Introduction and Planning, IBM DocumentGC34-5599-01.

Among many other capabilities, the brokers provide the capabilities to.

-   -   Route a message to several destinations using rules that act on        the contents of one or more fields in a message or message        header; and    -   Transform a message so that applications using different formats        can exchange messages in their own formats. The broker knows the        requirements of each application (for example, whether personal        names should have the surname first or last, with or without        middle initials, upper or lower case) and so it can transform        the message to the correct format (changing the order of fields,        the byte order, the language, and so on).

Within a message flow 40, the action to be taken can be definedaccording to the message structure, the message topic, or the datawithin the message. Alternatively, the originator of the message or itsdestination may be significant. Any combination of one or more of theseattributes can define the rules by which messages are processed. Amessage flow can process one message in several ways to deliver a numberof output messages, perhaps with different format and content, to anumber of target applications.

Each message flowing through a message broker has a specific structure,which is important and meaningful to the applications that send orreceive that message. Message structure information as used in IBM'sMQSeries Integrator products comprises a message domain, message set,message type, and wire format of the message. Together these valuesidentify the structure and format of the data the message contains.Every message flow that processes a message conforming to this structureand format must understand it to enable the message bit stream to beinterpreted.

The message structures which can be handled by the message brokerscomprise those which are predefined within one or more messagerepositories 60 as well as self-defining message structures (such asmessage data in XML which is a string of name-value pairs such theparser can simply locate a specified name in a message and extract thedata value). The message repositories hold information for a number ofmessage formats defining the arrangement of named data fields inmessages of the respective format. Then a format name lookup step canextract the message format template which can be applied to any messagehaving that format to generate a view of the message as a sequence ofname-value pairs which can be interpreted and manipulated.

When a message is to be processed by a message flow, it is firstnecessary to decode the message bit stream using one or more messageparsers 50 to enable the various components of the message to beunderstood. Message type and format information for messages predefinedin the message repositories is typically included in the messages'headers, and so a parser can recognise the message structure and formatwhen the message is received. Messaging products such as IBM's MQSeriesIntegrator product are known to include a number of message parsers 50which can parse known message structures including specific headers.However, with increasing systems and network integration, it is arequirement for these solutions to be extensible to add new parsers fornew message structures and new data formats within a message. Aparticular extensible solution which is implementable within computerprogram products such as IBM's MQSeries Integrator products, and whichsatisfies this message broker parsing requirement will now be describedin detail.

FIG. 2A shows an example message structure of a message as taken from aninput message queue. This comprises a message descriptor 100 (MD), oneor more additional message headers 110 (in this case RFH2), and amessage body 120. Normally the message data within the body part of themessage has a single format, and so a single parser will handle theparsing of the entire data contents. Nevertheless, as the bit stream ofthe message is analysed to identify the separate MD, headers and bodycomponents, each of these components will be assigned to an appropriateparser. Additionally, the message as modelled by parsing will include aProperties component which is generated from data provided by the inputnode of the message flow (see next paragraph—this is possible if onlymessages having certain properties are input to this input node) and/orfrom data extracted from the message headers. The Properties componentwill be handled by an appropriate parser which is typically differentfrom the parsers handling the MD, headers, and body. If the message bodycontains multiple nested formats, then multiple parsers will also berequired to parse the body. The implementation of parser selection andinvocation for handling nested message formats will be described below.

Within a simple example message flow of a message broker comprising aninput node, a processing node and an output node, the processing nodemay be a Compute node which transforms a message from one format toanother, so that sending and receiving applications can communicate witheach other using their own formats. Such a simple message flow is shownin FIG. 3. The input node requires the message to be parsed tounderstand its structure before the Compute node can perform its formattransformation.

On receipt of a message, the message broker 30 must pass the message bitstream (an array of bytes which make up the message) to a message parser50. Consider the previous example of an incoming message whichcomprises:

-   -   A message descriptor (MD)    -   An RFH2 header    -   An XML data portion

A parser selector 80 of the broker 30 sends the MD component to an MDparser 50, which accesses a structure template for MDs which is storedin the message repository 60 and applies this to the received MD tomodel it as a sequence of ordered name-value pairs. The MD parser thenreads one or more predefined message fields identifying the next messagecomponent's type and/or its format, compares this information with alist of component types/formats and selects and invokes another parser50 which has a predefined responsibility for handling parsing ofcomponents having this type and/or format. The MD parser also specifieswhat portion of the bit stream it has consumed to indicate where a nextselected parser 50 should begin. If the next component is the RFH2header, then this component is given to a specific RFH2 parser. The RFH2parser applies a stored RFH2 template to parse the header, modelling itas a sequence of name-value pairs, and then this RFH2 parser reads oneor more fields relating to the next component. If the next component isthe XML data portion, then the RFH2 parser invokes an XML parser inresponse to identifying that the component comprises XML. Since thetemplates stored for the MD and message headers determine what is thelast field within their respective message component, the MD and RFH2parsers can easily determine when to invoke the next parser.

Once the relevant parsers within the set of available parsers have beeninvoked to handle their respective chunks of the message bit stream andto create a syntax element tree, this syntax element tree can then bemanipulated by the message broker's processing nodes. For example, theformatting ‘Compute’ node constructs an output message the elements ofwhich can be defined using an SQL expression and are based on elementsof the input message. Other processing nodes' output message elementsmay be based on elements of the input message and data extracted from anexternal database.

The above description shows the sequence of operations of an embodimentof the present invention in which a selected parser selects and invokesa next parser for handling the next component of the message, but theabove description relates to an example with a single format messagebody. Many received messages will not be this simple.

Referring to FIG. 2B, if a message's body component 120 comprises anIDoc, then the MD 100 or another message header 110 will include anidentification of this component type (for example showing ‘IDOC’ in theformat field of the MD). This information is used during parsing of themessage to invoke a dedicated IDOC parser 50 and to indicate whichformat dictionary 60 is to be used. Thus far, there is no conceptualdifference from the XML body example in which selected parsers canselect and invoke further parsers. However, markup languages such as XMLare self-defining and so a single XML parser can be used. An IDoc, whichis not self-defining in the markup sense, also does not have apredicable number of data components. In particular, a control structureDC component 130 may be followed by one or many data segment DDcomponents 140. The last one of this sequence of DDs may be the end ofthe message, or it may be followed by another DC and one or more DDs, orthere may be additional non-IDoc message components. Therefore, ananalysis of the IDoc structure is necessary to determine which parser touse for which sub-component. In this case, it is desirable for the IDOCparser to control the selection and instantiation of particular parsersto handle subcomponents of the bit stream for which it isresponsible—the IDOC parser building an hierarchical tree of parsers.

In brief, the Version 4 IDoc structure comprises fixed size structures,the first of which is the DC control structure which is 524 bytes long.The one or more subsequent segment data DD's are each 1063 bytes long.Each DD structure is composed of name-value pairs plus a 1000 byte finalfield which holds the segment data. An IDoc can hold many differentmessage types and each message type can have many different segments.

The IDOC parser is invoked in response to identification of the IDOCtype information in a message descriptor or message header field.Firstly, the IDOC parser parses the DC component, applying its knowledgeof IDoc structures (a DC template) to generate a sequence of name-valuepairs. Next, the IDOC parser investigates the first field of the DDwhich follows the DC. This provides an identification of a specificmessage format within the dictionary of message formats which isassociated with the IDOC parser. This message format identifier anddictionary name is then passed to a Message Repository Manager whichboth implements the respective parser and holds the format templateswhich are required by the parser programs. This Message RepositoryManager comprises a collection of parsers and a dictionary of formats inwhich individual format names are associated with predefined formattemplates. The Message Repository Manager is thus instantiated to runthe relevant parser for the 1063 bytes of the current DD, to model thisdata for subsequent processing. The IDOC parser then investigates thefirst field of the next component of the message, which may be a DC orDD, and then instantiates the required parser to handle this nextcomponent. If it is a requirement for the message broker to be able tohandle messages which include additional data subsequent to the last DDof the IDoc, then the IDOC parser will also be required to investigatethe format of the next component to invoke an additional parser.

Thus, the IDOC parser takes input messages in valid IDoc format andcreates logical message tree structures which can be interpreted by thebroker's message flows. Similarly, this IDOC parser can take a logicalmessage tree created by a message flow and produce the data portion of amessage broker's message in IDoc format. That is, once a Compute node,for example, has finished manipulating the syntax element tree whichrepresents the message, the IDOC parser will be called once again torebuild the bit stream at the output node of the message broker'smessage flow.

In a particular implementation of the invention for parsing logicalmessage trees within a message broker, all such trees have a basicstructure which has a single high-level element known as the rootelement. The user data found in the body of the message has a high-levelelement which is one of the child elements (typically the last childelement) of the root element as well as additional elements at a lowerlevel of the hierarchy which correspond to the sub-components of theIDoc. For a SAP IDoc message, the name of the high level element is‘IDOC’ to match the message domain supported by the parser, since theIDOC parser registers with the broker for a message domain of ‘IDOC’.The IDOC parser then attempts to parse any input message that is read bythe message broker and has this message domain specified either in theRFH2 header or another message header or in the input node of theparticular message broker's message flow. The parser will thus create alogical message tree that reflects the contents of the message, with thetop-level element of the body having the name ‘IDOC’ and the rest of thedata for this IDoc structure built as a logical data structure unde thishigh-level element.

FIG. 3 provides a schematic representation of a SAP R/3 systemcommunicating with a message broker such as IBM's MQSeries Integratorproduct via a communication link and a link program component (such asthe MQSeries link for SAP R/3 program) which handles placing of amessage received from the SAP system onto a queue under transactionalcontrol (i.e. Using a transaction ID for a received message to ensureone full copy only of each message is placed on the queue). The inputnode of the message broker's message flow has been configured to readfrom this queue. The default message domain of the input node is set to‘IDOC’. A syntax element tree is generated by the parsing processes asdescribed above. An message including an IDoc sent from a SAP R/3 systemvia the link program component and then parsed as described above wouldbe represented by the parsing processes including the hierarchical IDOCparser as shown in FIG. 4. All is messages from the link programcomponent connecting the SAP R/3 system to the message broker include anadditional header not described above. This SAPH header is assigned toits own parser. As stated previously, any number of additional messageheaders and nested formats can be handled as described above. The linkprogram component also ensures that atomic transactional principles arefollowed when sending messages from the message broker to the SAPsystem, rolling back a message onto the broker's output queue to enablea retry if it is not possible to deliver it on a first attempt.

A list of field names of DC and DD structures which are understood bythe IDoc parser of a first implementation of the present invention areshown below. These are documented in the form in which they would beused in a SET statement of eSQL. For example:

SET OutputRoot.Properties = InputRoot.Properties; SET OutputRoot.MQMD =InputRoot.MQMD.Control Section (DC) FieldsAll fields must be specified and set

The syntax is <rootname>.<Parser Name>.<folder name>.<field name> = SET“OutputRoot”.“IDOC”.“DC”.“tabnam” = SET “OutputRoot”.“IDOC”.“DC”.“mandt”= SET “OutputRoot”.“IDOC”.“DC”.“docnum” = ‘0000000000000001’; SET“OutputRoot”.“IDOC”.“DC”.“docrel” = ‘45B’; SET“OutputRoot”.“IDOC”.“DC”.“status” = ‘30’; SET“OutputRoot”.“IDOC”.“DC”.“direct” = ‘1’; SET“OutputRoot”.“IDOC”.“DC”.“outmod” = ‘4’; SET“OutputRoot”.“IDOC”.“DC”.“exprss” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“test” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“idoctyp” = ‘MATMAS01’; SET“OutputRoot”.“IDOC”.“DC”.“cimtyp” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“mestyp” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“mescod” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“mesfct” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“std” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“stdvrs” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“stdmes” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“sndpor” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“sndprt” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“sndpfc” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“sndprn” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“sndsad” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“sndlad” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“rcvpor” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“rcvprt” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“rcvpfc” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“rcvprn” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“rcvsad” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“rcvlad” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“credat” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“cretim” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“refint” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“refgrp” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“refmes” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“arckey” = ‘’; SET“OutputRoot”.“IDOC”.“DC”.“serial” = ‘’;Data Section (DD) Fields

To access each DD segment use the array suffix ie. DD [1], DD[2] etc.

Note the use of the ‘2’ sufix to give unique field names on the mandtand docnum fields

SET OutputRoot.IDOC.DD[I].segnam = ‘E2MAKTM001’; SETOutputRoot.IDOC.DD[I].mandt2 = ‘111’; SET OutputRoot.IDOC.DD[I].docnum2= ‘9999999999999111’; SET OutputRoot.IDOC.DD[I].segnum = ‘111000’; SETOutputRoot.IDOC.DD[I].psgnum = ‘000111’; SETOutputRoot.IDOC.DD[I].hlevel = ‘11’;Segment Fields

Use the ‘sdatatag’ keyword to indicate to the parser that it is theelement that contains the segment data which is to be manipulated. TheMRM indicates that the MRM will handle the transformation. The finalfield identifies the actual field of the segment. The final line is thefiller for the segment as an incoming IDoc to SAP must have each segment1000 bytes long.

SET OutputRoot.IDOC.DD[I].sdatatag.MRM.msgfn = ‘006’; SETOutputRoot.IDOC.DD[I].sdatatag.MRM.spras = ‘E’; SETOutputRoot.IDOC.DD[I].sdatatag.MRM.maktx = ‘Buzzing all night’; SETOutputRoot.IDOC.DD[I].sdatatag.MRM.msgfn = ‘006’; SETOutputRoot.IDOC.DD[I].sdatatag.MRM.spras_iso = ‘EN’; SETOutputRoot.IDOC.DD[I].sdatatag.MRM.fill954 = ‘ ’;

Note that the traditional way of parsing a bit stream is to traverse theincoming bit stream from left to right, paring the respective sectionsof the bit stream to the appropriate parser until all the bit stream isparsed. This in effect is a horizontal pass of the data. According tothe embodiment of the present invention described above, an hierarchicaltree of parsers may be created since one or more of the selected parsersis able to call other parsers to process subcomponents of the portion ofthe message bit stream it is handling, or to call a parser to handle thenext component of the message subsequent to that portion. Thus, anincoming bit stream is passed to an appropriate parser and this parserthen passes sections of this bit stream to other parsers beforereturning back to the original bit stream to have the remains of theunclaimed bit stream passed to the next appropriate parser. Thishierarchical parsing approach allows a message broker to model datawhich is not known until runtime, without excessive communication flowsto and from a single selector process and without the overhead of havingto update a generic parser or generic selector each time a new dataformat is added. This approach enables a dedicated parser to be givendetailed knowledge of the structures and formats of a particular type ofmessage for which it will be responsible. Implementing this level ofdetailed knowledge of many different message types in a single parser orparser-selector would be a very inefficient.

While an embodiment of the present invention has been described aboveusing the example of IDocs, aspects of the invention are equallyapplicable to other message data formats such as messages which includedata in the SWIFT message format where the sequence of components is notfixed or other message structures and formats which are not fullypredefined.

The invention is also applicable to data processing environments otherthan message brokers in an heterogeneous network, although the inventionis especially advantageous for that environment. Thus, one aspect of thepresent invention provides a parsing program comprising a set ofselectable parsers which are each adapted for parsing a specific set ofone or more data formats, wherein a plurality of the set of parsers havethe following capabilities:

-   -   Parsing a first component of a bit stream;    -   Identifying the data format of a second component of the bit        stream, which may be a subcomponent of the first component;    -   In response to the format identification, selecting and invoking        another one of the set of parsers to parse the second component.

This enables a selected parser of a set to control selection of otherparsers to handle either subcomponents of the portion of a bit streamfor which it has been given responsibility, or to invoke a next parserfor a component of the bit stream subsequent to its assigned portion ofthe bit stream, both of which capabilities provide increased efficiencywhen parsing a bit stream which includes multiple nested data formats.

The steps of a method according to an embodiment of the invention forparsing a data stream including multiple nested data formats can besummarised with reference to FIG. 5. This shows an initial step of aselector program component 80 identifying 200 the format of a first datacomponent of the bit stream. The bit stream may include a fieldcontaining the class name of a specific parser for handling this dataformat or it may include data format information that can be comparedwith a repository holding information regarding which parsers areregistered to handle which formats. The information in the repositorymay be parser class names. The selector then instantiates 200 the namedparser class to generate a specific instance of the appropriate parserto handle the current format. In alternative embodiments, the format ofthe first portion of all bit streams may be predefined for theparticular system so that the appropriate first parser can be invokedwithout format analysis. The selected parser begins parsing 210 thefirst component of the bit stream.

Two possible situations can now arise. If the bit stream has a singleformat, the selected parser parses the whole bit stream to output amodel 230 of the data for subsequent processing. If the bit streamincludes multiple formats, the first selected parser is responsible foridentifying 240 the format of the component of the bit stream whichfollows the component it has parsed. This involves the parser analysingthe contents of specific fields of the bit stream in a similar manner tothe initial analysis performed by the selector component 80. The currentparser then calls a create parser method 240 for the appropriate parserclass to invoke a parser to handle this next component of the bitstream. This parser parses its component 210, and the selection andparsing steps are repeated for subsequent components having differentformats. Each parser reports what portion of the bit stream its parsinghas handled so that the information is available to determine where thenext selected parser should begin.

While this method is advantageous for a bit stream comprising distinctlyseparable components, it is especially advantageous to enable selectedparsers to invoke a next parser when one component includes adifferently formatted subcomponent such that a parser handling thecomponent only requires a different parser to handle its subcomponentand then requires control to be returned to it. In this case, a selectedparser is handed a chunk of the bit stream having a known size, itparses this chunk and returns control to the parser which selected it.This provides significant advantages compared with known alternativeswhen it is necessary to parse a component in order to determine theformat of (and hence select a suitable parser for) its subcomponents.

1. A message processing system including: a set of selectable parsers,each selectable parser being adapted for analyzing a respective set ofmessage data formats and being selectable in response to identifying amessage data format within the respective set; a process for invoking,by a data processing apparatus in the message processing system, aparser from the set, wherein at least one of said selectable parsers:parses a first component of a message having a message data formatwithin the respective set; identifies the data format of a secondcomponent of the message; responsive to said identification, selectsanother one of said set of parsers and for invoking the selected parserto parse the second message component; and includes a repository ofmessage format templates, each template representing the field structureof a particular format of message component, wherein said at least oneparser is adapted to parse the first component and then to read a formatfield of a second component of the message to identify the data formatof the second component, and wherein invoking the selected parser toparse the second component includes providing to the selected parser anidentification of the message format template corresponding to theidentified format.
 2. A message processing system according to claim 1,wherein said at least one parser parses the first component to enableidentification of a format field containing information relating to theformat of the second component.
 3. A parser program stored on anon-transitory computer readable medium comprising program code forexecution on a computer for processing a bit stream which may contain aplurality of data formats, the parser program including a set ofselectable parsers, each adapted for analyzing a specific set of one ormore data formats, wherein the parser program includes: instructions forselecting, by a parser selector, a first parser to parse a firstcomponent of a bit stream; instructions for parsing, by the firstparser, the first component of the bit stream; instructions foridentifying, by the first parser, the data format of a second componentof the bit stream; and instructions for selecting, by the first parser,a second parser in the set of parsers based on the data format of thesecond component and for invoking the second parser to parse the secondcomponent, wherein said instructions for invoking the second parserincludes instructions for inputting the second component to the secondparser.
 4. The parser program according to claim 3, wherein the secondparser is adapted to access a format template from a format dictionarycorresponding to the format identified by the first parser.
 5. Theparser program according to claim 3, wherein the instructions forparsing the first component are adapted to output a name-value pairindicating the format of the second component.
 6. The parser programaccording to claim 3, wherein the instructions for identifying the dataformat of the second component comprises instructions for analyzing aformat field of the second component.
 7. The parser program according toclaim 3, wherein the second parser returns control to the first parserafter parsing the second component.
 8. The parser program according toclaim 3, further comprising: instructions for identifying, by the secondparser, a data format of a third component of the bit stream; andinstructions for selecting, by the second parser, a third parser in theset of parsers based on the identified data format of the thirdcomponent and for invoking the third parser to parse the thirdcomponent.
 9. A computer implemented method of parsing a messagecontaining a plurality of data formats, the computer implemented methodcomprising: responsive to identifying a data format of a first componentof the message, selecting and invoking, by a data processing apparatus,a first parser to parse the first component; identifying the data formatof a second component of the message using the first parser; andresponsive to said identification, using the first selected parser toselect and invoke, based on the data format of the second component, asecond parser for parsing the second message component, wherein invokingthe second parser includes inputting the second component to the secondparser; and parsing the second component using the second parser. 10.The computer implemented method according to claim 9, wherein the secondparser is adapted to access a format template from a format dictionarycorresponding to the format identified by the first parser.
 11. Thecomputer implemented method according to claim 9, wherein parsing thefirst component outputs a name-value pair indicating the format of thesecond component.
 12. The computer implemented method according to claim9, wherein identifying the data format of the second component comprisesanalyzing a format field of the second component.
 13. The computerimplemented method according to claim 9, wherein the second parserreturns control to the first parser after parsing the second component.14. The computer implemented method according to claim 9, furthercomprising: identifying, by the second parser, a data format of a thirdcomponent of the bit stream; and selecting, by the second parser, athird parser in the set of parsers based on the identified data formatof the third component and for invoking the third parser to parse thethird component.